Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing
Identifieur interne : 000452 ( Main/Exploration ); précédent : 000451; suivant : 000453Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing
Auteurs : Gabriella Kazai [Royaume-Uni] ; Marijn Koolen [Pays-Bas] ; Jaap Kamps [Pays-Bas] ; Antoine Doucet [France] ; Monica Landoni [Suisse]Source :
- Lecture Notes in Computer Science [ 0302-9743 ] ; 2011.
Abstract
Abstract: The goal of the INEX Book Track is to evaluate approaches for supporting users in searching, navigating and reading the full texts of digitized books. The investigation is focused around four tasks: 1) Best Books to Reference, 2) Prove It, 3) Structure Extraction, and 4) Active Reading. In this paper, we report on the setup and the results of these tasks in 2010. The main outcome of the track lies in the changes to the methodology for constructing the test collection for the evaluation of the Best Books and Prove It search tasks. In an effort to scale up the evaluation, we explored the use of crowdsourcing both to create the test topics and then to gather the relevance labels for the topics over a corpus of 50k digitized books. The resulting test collection construction methodology combines editorial judgments contributed by INEX participants with crowdsourced relevance labels. We provide an analysis of the crowdsourced data and conclude that – with appropriate task design – crowdsourcing does provide a suitable framework for the evaluation of book search approaches.
Url:
DOI: 10.1007/978-3-642-23577-1_9
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 001353
- to stream Istex, to step Curation: 001274
- to stream Istex, to step Checkpoint: 000109
- to stream Main, to step Merge: 000458
- to stream Main, to step Curation: 000452
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing</title><author><name sortKey="Kazai, Gabriella" sort="Kazai, Gabriella" uniqKey="Kazai G" first="Gabriella" last="Kazai">Gabriella Kazai</name></author><author><name sortKey="Koolen, Marijn" sort="Koolen, Marijn" uniqKey="Koolen M" first="Marijn" last="Koolen">Marijn Koolen</name></author><author><name sortKey="Kamps, Jaap" sort="Kamps, Jaap" uniqKey="Kamps J" first="Jaap" last="Kamps">Jaap Kamps</name></author><author><name sortKey="Doucet, Antoine" sort="Doucet, Antoine" uniqKey="Doucet A" first="Antoine" last="Doucet">Antoine Doucet</name></author><author><name sortKey="Landoni, Monica" sort="Landoni, Monica" uniqKey="Landoni M" first="Monica" last="Landoni">Monica Landoni</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:1AF84F943F9B0E956726A6E1841A58E283104B1E</idno><date when="2011" year="2011">2011</date><idno type="doi">10.1007/978-3-642-23577-1_9</idno><idno type="url">https://api.istex.fr/document/1AF84F943F9B0E956726A6E1841A58E283104B1E/fulltext/pdf</idno><idno type="wicri:Area/Istex/Corpus">001353</idno><idno type="wicri:Area/Istex/Curation">001274</idno><idno type="wicri:Area/Istex/Checkpoint">000109</idno><idno type="wicri:doubleKey">0302-9743:2011:Kazai G:overview:of:the</idno><idno type="wicri:Area/Main/Merge">000458</idno><idno type="wicri:Area/Main/Curation">000452</idno><idno type="wicri:Area/Main/Exploration">000452</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing</title><author><name sortKey="Kazai, Gabriella" sort="Kazai, Gabriella" uniqKey="Kazai G" first="Gabriella" last="Kazai">Gabriella Kazai</name><affiliation wicri:level="1"><country xml:lang="fr">Royaume-Uni</country><wicri:regionArea>Microsoft Research</wicri:regionArea></affiliation><affiliation><wicri:noCountry code="no comma">E-mail: v-gabkaz@microsoft.com</wicri:noCountry></affiliation></author><author><name sortKey="Koolen, Marijn" sort="Koolen, Marijn" uniqKey="Koolen M" first="Marijn" last="Koolen">Marijn Koolen</name><affiliation wicri:level="1"><country xml:lang="fr">Pays-Bas</country><wicri:regionArea>University of Amsterdam</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Pays-Bas</country></affiliation></author><author><name sortKey="Kamps, Jaap" sort="Kamps, Jaap" uniqKey="Kamps J" first="Jaap" last="Kamps">Jaap Kamps</name><affiliation wicri:level="1"><country xml:lang="fr">Pays-Bas</country><wicri:regionArea>University of Amsterdam</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Pays-Bas</country></affiliation></author><author><name sortKey="Doucet, Antoine" sort="Doucet, Antoine" uniqKey="Doucet A" first="Antoine" last="Doucet">Antoine Doucet</name><affiliation wicri:level="1"><country xml:lang="fr">France</country><wicri:regionArea>University of Caen</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">France</country></affiliation></author><author><name sortKey="Landoni, Monica" sort="Landoni, Monica" uniqKey="Landoni M" first="Monica" last="Landoni">Monica Landoni</name><affiliation wicri:level="1"><country xml:lang="fr">Suisse</country><wicri:regionArea>University of Lugano</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Suisse</country></affiliation></author></analytic><monogr></monogr><series><title level="s">Lecture Notes in Computer Science</title><imprint><date>2011</date></imprint><idno type="ISSN">0302-9743</idno><idno type="eISSN">1611-3349</idno><idno type="ISSN">0302-9743</idno></series><idno type="istex">1AF84F943F9B0E956726A6E1841A58E283104B1E</idno><idno type="DOI">10.1007/978-3-642-23577-1_9</idno><idno type="ChapterID">9</idno><idno type="ChapterID">Chap9</idno></biblStruct></sourceDesc><seriesStmt><idno type="ISSN">0302-9743</idno></seriesStmt></fileDesc><profileDesc><textClass></textClass><langUsage><language ident="en">en</language></langUsage></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: The goal of the INEX Book Track is to evaluate approaches for supporting users in searching, navigating and reading the full texts of digitized books. The investigation is focused around four tasks: 1) Best Books to Reference, 2) Prove It, 3) Structure Extraction, and 4) Active Reading. In this paper, we report on the setup and the results of these tasks in 2010. The main outcome of the track lies in the changes to the methodology for constructing the test collection for the evaluation of the Best Books and Prove It search tasks. In an effort to scale up the evaluation, we explored the use of crowdsourcing both to create the test topics and then to gather the relevance labels for the topics over a corpus of 50k digitized books. The resulting test collection construction methodology combines editorial judgments contributed by INEX participants with crowdsourced relevance labels. We provide an analysis of the crowdsourced data and conclude that – with appropriate task design – crowdsourcing does provide a suitable framework for the evaluation of book search approaches.</div></front></TEI><affiliations><list><country><li>France</li><li>Pays-Bas</li><li>Royaume-Uni</li><li>Suisse</li></country></list><tree><country name="Royaume-Uni"><noRegion><name sortKey="Kazai, Gabriella" sort="Kazai, Gabriella" uniqKey="Kazai G" first="Gabriella" last="Kazai">Gabriella Kazai</name></noRegion></country><country name="Pays-Bas"><noRegion><name sortKey="Koolen, Marijn" sort="Koolen, Marijn" uniqKey="Koolen M" first="Marijn" last="Koolen">Marijn Koolen</name></noRegion><name sortKey="Kamps, Jaap" sort="Kamps, Jaap" uniqKey="Kamps J" first="Jaap" last="Kamps">Jaap Kamps</name><name sortKey="Kamps, Jaap" sort="Kamps, Jaap" uniqKey="Kamps J" first="Jaap" last="Kamps">Jaap Kamps</name><name sortKey="Koolen, Marijn" sort="Koolen, Marijn" uniqKey="Koolen M" first="Marijn" last="Koolen">Marijn Koolen</name></country><country name="France"><noRegion><name sortKey="Doucet, Antoine" sort="Doucet, Antoine" uniqKey="Doucet A" first="Antoine" last="Doucet">Antoine Doucet</name></noRegion><name sortKey="Doucet, Antoine" sort="Doucet, Antoine" uniqKey="Doucet A" first="Antoine" last="Doucet">Antoine Doucet</name></country><country name="Suisse"><noRegion><name sortKey="Landoni, Monica" sort="Landoni, Monica" uniqKey="Landoni M" first="Monica" last="Landoni">Monica Landoni</name></noRegion><name sortKey="Landoni, Monica" sort="Landoni, Monica" uniqKey="Landoni M" first="Monica" last="Landoni">Monica Landoni</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000452 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000452 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Ticri/CIDE
|area= OcrV1
|flux= Main
|étape= Exploration
|type= RBID
|clé= ISTEX:1AF84F943F9B0E956726A6E1841A58E283104B1E
|texte= Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing
}}
| This area was generated with Dilib version V0.6.32. |  |